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BACKGROUND 


A 2015 survey from Wolters Kluwer’ showed that only 14% of re- 
spondents on behalf of law schools were aggressively planning for online 


+ Victoria Sutton, MPA, PhD, JD, is the Paul Whitfield Horn Professor, Texas Tech Uni- 
versity School of Law. This research was made possible by a grant from Worldwide E-Learn- 
ing, Office of the Provost, Texas Tech University. 1 would like to also thank the faculties of 
Texas Tech University School of Law and the University of Houston School of Law for lis- 
tening and making meaningful comments on presentations of the findings in this research. 

1. Wolters Kluwer published a report, “The Leading Edge Report,” which they describe 
as “intended to succinctly summarize many of the key areas of focus. for law schools today 
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legal education, while 19% did not plan to ever offer online education.” 
The rest of the responses fell into the proceeding-with-caution category.’ 
The authors attribute two major factors to this state of affairs—first, the 
limited number of hours allowed for online courses by the American Bar 
Association (ABA); and second, the technological difficulties and the re- 
luctance of professors to learn the technology necessary to offer online 
courses.’ In addition, what the report calls a “greater barrier over time” 
was the general “wide-scale perception that online education is worth less 
than on-campus education.” Online offerings in law schools have in- 
creased significantly in the last four years since the Wolters Kluwer sur- 
vey. By February 2018, ABA had accredited three hybrid, online J.D. 
programs,’ and ABA responded with more than doubling the amount of 
online credit that could be earned in a Juris Doctorate degree (J.D.), revis- 
ing the ABA Standard in August 2018.8 

The question of whether online, or e-learning, education is “worth 
less” than the traditional classroom experience had not been empirically 
tested, before this study; and the idea that online education was “worth 
less” was being passed along as a kind of folk psychology among legal 
educators.’ This study was designed to produce evidence-based answers 


seeking to evolve their teaching model to meet the challenges and opportunities of 21st cen- 
tury legal education.” Legal Education Leading Edge Report, 2015 Edition, WOLTERS 
KLUWER, http://www.wklegaledu.com/supplements/id-9781454875307/Legal_Educa- 
tion_Leading Edge Report_2015 Edition (last visited Sept. 28, 2019). 

2. Leading Edge Rep. (Wolters Kluwer) 19 (2015) [hereinafter Leading Edge]. 

3. Id. at 19 fig.4. 


Response to the question: “Many law schools are considering expanding the num- 
ber of online course options their students have access to. Such online courses would 
involve students meeting faculty ‘virtually’ at specified times for discussions of study 
materials, accessing pre-recorded lectures, and downloading assignments and papers 
online. Which statement best describes your law school’s plans for including online 
courses in its curriculum?” The survey was conducted in May 2014 and was com- 
pleted by 40 respondents. Of these, 36 answered this question. 


Id. Where 19% answered “[w]e do not plan to offer”; 67% the majority, answered, “[w]e plan 
to move cautiously”; and only 14% responded, “[w]e plan to be aggressive.” Id. 

4, Id. at 18-19. 

5. Id. 

6. Yvonne M. Dutton, Margaret Ryznar & Kayleigh Long, Assessing Online Learning 
in Law Schools: Students Say Online Classes Deliver, 96 DENV. L. REV. 493, 494 (2019). 

7. Henry Kronk, The Ice is Melting for Hybrid J.D. Programs, ELEARNING INSIDE (Feb. 
22, 2018), https://news.elearninginside.com/the-ice-is-melting-for-hybrid-j-d-programs/. 

8. Barry A. Currier, Adoption and Implementation of Revised ABA Standards and Rules 
of Procedure for Approval of Law Schools, 2018 A.B.A. SEC. OF LEGAL EDUC. & ADMISSIONS 
TO THE BAR. 

9. Leading Edge, supra note 3, at 18-19. 
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to these questions. 

Research was lacking; however, there is good reason for the lack of 
research data—there are just a very small number of asynchronous law e- 
learning courses available to study, and there is additional work required 
to do it properly.'® So in deciding whether to plunge into the asynchronous 
law teaching domain, without such evidence, it seemed inevitable that I 
should develop a comparative study and make a contribution to this need 
for the legal academy. 

Returning to the Wolters-Kluwer survey, perhaps the right question 
to be asked next, is not whether online legal education is better or worse; 
but how does it compare to traditional law course experiences? Saying an 
apple is better than an orange is not that useful, anyway. But describing 
whether an orange is more difficult to peel than an apple, for example, 
could provide more useful information in making decisions about how to 
go about engaging with oranges and apples. The same can be said about 
asynchronous courses and traditional law courses. 

So in Fall 2015, when I designed and taught my law school’s first 
completely asynchronous doctrinal course, I decided to try to begin to fill 
the gap of empirical knowledge on e-learning in law schools and to for- 
mally conduct an experiment to determine how the asynchronous, e-learn- 
ing model in legal education compared to “traditional” law courses.'' It 
would be the first such study. 

The term, traditional law courses, would include courses using the 
methods prescribed by Langdell, and the tradition of law school courses 
for more than one hundred years.'* This comparative data is particularly 
valuable given the growing criticism of this long-established law teaching 
method for failing to meet the creative needs to prepare students to be 
practicing lawyers of today and tomorrow.'? Some have also noted re- 
sistance to changing that method, or simply that nothing has successfully 


10. Victoria Sutton, Asynchronous E-Learning in Legal Education: A Comparative Study 
1 (Aug. 6. 2016) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?ab- 
stract_id=2819034. 

ll. Jd. at 2. 

12. Susan Sturm & Lani Guinier, The Law School Matrix: Reforming Legal Education in 
a Culture of Competition and Conformity, 60 VAND. L. Rev. 515, 515-16, 532 (2007). 

13. Id. at 515-16. 


Many reformers agree that the prevailing law school model developed in the nine- 
teenth century does not adequately prepare students to become effective twenty-first 
century lawyers. Langdell’s case method, designed around private domestic law, ap- 
pellate cases, and the Socratic method, increasingly fails to teach students “how to 
think like a lawyer” in the world students will occupy. The curriculum over-empha- 
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replaced it, writing that “[s]ince Langdell, there has been no systematic 
effort to realign the theory of law and the concept of the profession with 
the basic design of the law school as an institution.” Given this sum- 
mary of the state of legal education, an empirical study of e-learning as a 
platform for legal education seemed a gaping hole in information neces- 
sary to make decisions about e-learning. 


I. DESIGNING THE DOCTRINAL LAW COURSE FOR ASYNCHRONOUS, 
ONLINE DELIVERY 


There is a paucity of literature on designing asynchronous law 
courses, as well as empirical data about them. "° One source proved that 
the literature does exist, a brief, seventy-four page book by Jennifer 
Camero, Teaching Law Online.'® Advice ranged from cautionary tales to 
how to discussion of the common Blackboard® platform for e-learning.'’ 
Much of the information was drawn from non-law courses, but was very 
helpful as a starting point.'® 

My objective in designing this course was to develop an online ex- 
perience for students that was noticeably different from a traditional law 
class, which might make the experience at least comparable to a tradi- 
tional course, when factoring in the “tradeoffs” that are inevitable.'? For 
example, producing video-lectures utilizing editing and cuts to relevant 
visuals could be used to break the monotony of the “talking head” video,” 
Another goal toward that objective was to develop more points of contact 


sizes adjudication and discounts many of the important global, transactional, and fa- 
cilitative dimensions of legal practice. Law school has too little to do with what law- 
yers actually do and develops too little of the institutional, interpersonal, and investi- 
gative capacities that good lawyering requires. The Socratic method in the large 
classroom, though valuable as a way to teach sharp analytic skills, is ill-suited to fos- 
tering “legal imagination,” which is what lawyers need most to become effective ad- 
vocates, institutional designers, transaction engineers, and leaders. It also contributes 
to law student disengagement, particularly for women and people of color. 


Id. 

14. Id. at 549. 

15. Sutton, supra note 10, at 2. 

16. See generally JENNIFER CAMERO, TEACHING LAW ONLINE (2015) (discussing the tran- 
sition from classroom to cyberspace). 

17. See generally id. (discussing the ABCs of developing an online course). 

18. See id. at 43-51 (listing non-law related resources that the author relied upon in her 
writing). 

19. Sutton, supra note 10, at 2. 

20. Bill Golden, Why You Need to Rethink the Talking-Head Video: 4 Tips to Improve 
Video Engagement, TENDO (Oct. 22, 2018), https://tendocom.com/blog/why-you-need-to-re- 
think-the-talking-head-web-video/. 
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with the student through e-methods, to replace the traditional, two or three 
times a week face-to-face meetings in a classroom.” A third goal toward 
my objective for the course was to simulate the Socratic method by giving 
frequent feedback to the student using technology to facilitate that process 
at a much greater rate and volume using automatic grading on objective 
tests, which provides faster feedback than one could do using traditional 
methods of paper or even “scantron” grading.” Further, I wanted to test 
as many of the ABA Standards for online education?’ as possible with 
this course as part of the experiment’s comparative assessment.”4 


Il. RELEVANT GUIDANCE, RULES & REGULATIONS FOR THE DESIGN & 
ADMINISTRATION OF AN ONLINE COURSE 


There are four documents for compliance and conformance purposes 
that necessarily had to be consulted in designing and administering a 
course: (1) ABA Standard 306; (2) University e-learning standards; (3) 
Working Group for Distance Learning for Legal Education (WGDLLE) 
draft guidance on best practices in online education, the law professors ad 
hoc Working Group for online law teaching best practices; and (4) indi- 
vidual law school faculty rules for e-learning.”> This task requires contin- 
ual review to ensure all required syllabi language, all rubrics and all 
standards are met at every point.” In some cases, the standards for e- 
learning were more restrictive than for a traditional course.”’ 

For example, where the take-home exam has been a staple of law 
school testing for decades, ABA Standard 306(f) (previously 306(g)) re- 
quires that for an e-learning course a law school, “shall establish an ef- 
fective process for verifying the identity of students taking distance edu- 
cation courses.””* This is a challenge for all disciplines and universities 
which are hiring costly proctor programs that literally watch students on 
their laptop camera (some using video for later viewing) while each of 
them takes the exam.” Some simply give up on the idea of asynchronous 


21. Sutton, supra note 10, at 2. 

22. Id. at 2-3. 

23. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW SCHS. § 306 at 19 (AM. 
BAR Ass’N 2019). 

24. Sutton, supra note 10, at 3. 

25. Id. 

26. Id. 

27. Id. 

28. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW SCHS. § 306 at 19 (AM. 
BAR Ass’N 2019). 

29. See Anne Eisenberg, Keeping an Eye On Online Test-Takers, N.Y. TIMES, Mar. 2, 
2013 at4. 
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learning for the examination and they require students take a final exam 
in the traditional way, by coming to a controlled, proctored classroom at 
the appointed time.*° However, in this course, another goal was to achieve 
compliance with ABA Standard 306(g) (now 306(f)) while still keeping 
the course asynchronous.”! 

Each university, with a law school, has standards for language for 
the syllabus that includes such issues as items to be purchased by the stu- 
dent, like a laptop camera.” These best practices rules may also require 
that students write a computer emergency plan for participating in the 
course.*? In addition, every state where online education is offered has its 
own regulatory approval process, which is important if you are recruiting 
students for e-learning, nationwide.** In addition, compliance issues with 
the Americans with Disabilities Act specific to e-learning are also part of 
the university guidance.” There are potentially other requirements which 
e-learning must comply with, including the Digital Millennium Copy- 
right Act.*° 

The law school ad hoc working group, Work Group for Distance 
Learning for Legal Education (WGDLLE), worked from 2011-2015 to 
establish the first “best practices” document which is immensely helpful 
in designing a law course and the online tools that can be utilized.*” For 
example, the report recommends lectures being divided into five to seven 
minute sessions.’ Technology has made some of its conclusions obso- 
lete, like the lack of interactivity with online pre-recorded lectures.” 
Newer technology has made interactivity with pre-recorded lectures pos- 
sible.*° 

Finally, your own law school faculty rules may not yet have rules for 


30. Id. 

31. Sutton, supra note 10, at 3. 

32. Id. 

33. Id. 

34. Id. 

35. Id. 

36. See Digital Millennium Copyright Act of 1998, Pub. L. No. 105-304, 112 Stat. 2860 


37. Best Prac. Recommendations of Distance Learning for Legal Educ. 2.0, WORK 
GROUP OF DISTANCE LEARNING FOR LEGAL EDUCATION, at 2, 
http:/Awww.wedlle.org/files/2015/03/BestPracticeR ecommendationsforDistanceLearning- 
forLegalEducation-2015.pdf (last visited Sept. 28, 2019) [hereinafter Best Prac.]. 

38. Id. at 14. 

39. Id. at21. 

40. Id. at 28, An example ofa relatively new platform that allows inserting questions into 
a pre-recorded video to make it interactive during the lecture is edpuzzle. See Make any video 
your lesson, EDPUZZLE, http://edpuzzle.com (last visited Sept. 28, 2019). 
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e-learning courses, but if they do, it may include important information 
about teaching load credit for e-learning courses, minutes of video-lec- 
ture and allowable or disallowed practices.*! Another important issue 
with the ABA Standards is the requirement for a specific number of lec- 
ture minutes, as well as your own law school rules for how many minutes 
constitute the ABA “credit hour.”® For example, ABA Standard 310 (b) 
specifies that 


A “credit hour” is an amount of work that reasonably approximates: (1) 
not less than one hour of classroom or direct faculty instruction... or 
(2) at least an equivalent amount of work as required in subparagraph 
(1) of this definition for other academic activities as established by the 
institution, including simulation, field placement, clinical, co-curricu- 
lar, and other academic work leading to the award of credit hours.” 


To comply, the number of minutes for lecturing must make up the 
credit hour; for example, fifty minutes of lecture can be defined as your 
school’s credit hour.** However, the list of “other academic activities” 
should include online activities like discussion board, quizzes and other 
online activities.*° The rule allows “other academic activities as estab- 
lished by the institution,” which would require a law school rule to at least 
list the kinds or scope of activities used in online courses to comply with 
this standard.*° One broad definition for a law school rule might be that 
academic activities not unlike activities you might do in a face-to-face 
class would cover quizzes, discussion board and many of the online tools. 

Since teaching online differs significantly from the traditional ex- 
change of ideas ina classroom, a fifty-minute video lecture can result in 
potentially much more information being covered in a video-lecture with 
no student discussion (which would be typical of a law school class).*’ 
This can result in covering more material than should be covered in a 
three-hour course, for example.*® Instead, considering time for discussion 
boards, quizzes, e-journal writing, and additional video clips should be 
included in the definition of class minutes, particularly because these are 


41. Sutton, supra note 10, at 3. 

42. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW SCHS. § 310 at 21 (AM. 
Bar Ass’N 2019). 

43. Id. at § 310(b)(2). 

44. See id. at § 310. ABA Standard 310, Interpretation 310-1 “[flor purposes of this 
Standard, fifty minutes suffices for one hour of classroom or direct faculty instruction.” Id. 

45. Id. at § 310(b) at 21. 

46. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW Scus. § 310(b)(2) at 
21 (Am. Bar Ass’N 2019). 

47. Sutton, supra note 10, at 3. 

48. Id. at 3-4. 
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activities that tend to replace the Socratic exchange of ideas in a class- 
room.” Using the fifty-minute lecture option in ABA Standard 310(b)(1), 
from my own anecdotal observation, will inevitably result in the equiva- 
lent of a four-hour course, rather than a three-hour course.*° Thus, the use 
of ABA Standard 310(b)(2) as adopted to your own law school standards 
fits the needs of online courses much better than the ABA Standard 
310(b)(1) requirement.*! 


IH. DESIGNING THE ONLINE COURSE 


One of the objectives in designing this course was to create an inter- 
active or modified Socratic method online. A standard approach that could 
be repeated each week was also important to allow students to focus on 
the substance of the course, rather than navigating an unpredictable 
weekly format. 

With those design objectives in mind, the design of the course had a 
weekly, repeating routine of assignments.” The basic work for the week 
looked like Figure 1.°? Figure 2 shows the cyclical weekly pattern of as- 
signments with additional assessments. 


BRS Week 


The ABA Standard 314 requires assessment methods for all courses, 
including formative and summative assessments, but does not require that 
all of them be used in every course.” This relatively new standard re- 
quires additional assessment beyond the one final exam, which has been 


49. Id. at4. 
50. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW ScHS. § 310(b)(1) at 


51. Jd. at § 310(b) at 21. 

52.. Sutton, supra note 10, at 4. 

53. dd. 

54. Id. 

55. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW SCHS. § 314 at 23 (AM. 
Bar Ass’n 2019). 
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the traditional legal teaching assessment method for decades.°° E-learn- 
ing is organically designed to meet this criteria because continuing as- 
sessment of students is important in an asynchronous course.*’ For pur- 
poses of this research several assessment methods were used that were 
not traditional.: The video-project was probably the least traditional of 
all the assessment methods, followed by the more standard Discussion 
Board, and a weekly E-Journal for recording observations about the 
week’s lectures, reading, or assignments.” The Discussion Board al- 
lowed sharing thoughts with the entire class while the E-Journal allowed 
sharing only between the student and the professor. There were several 
writing assignments including one group, memorandum writing assign- 
ment.°! The traditional mid-term and final examination were also in- 
cluded as assessment tools. During the course of the semester, lectures 
were ten to fifteen minutes long each followed by a short, five question 
multiple choice quiz, timed for twenty minutes. Each quiz was to be 
taken at the end of each corresponding video-lecture.“ A numbered, in- 
dexing system was used to match the quizzes with the video-lectures.” 
The organization of the materials on the Blackboard® platform uti- 
lized this numbered index system to organize video-lecture, reading, and 
quizzes. Each reading assignment, video-lecture, and quiz all shared the 
same index number, developed for the course to keep the vast amount of 
material organized on the online platform for the students as well as for 
the professor.®’ The index system worked as follows. An index number 
was designed based on a three-hour course that would meet three times a 
week for fifty minutes.°* The week would be designated by the first num- 
ber (one to fifteen weeks for the semester). The second number would 


56. See Jacob Wentzel, New ABA Requirements Bring Changes to Law School Class- 
rooms, Creating Opportunity, and Chaos, NULR oF NoTE (Dec. 1, 2017), blog.northwest- 
ernlaw.review/?p=214. 

57. Best Prac., supra note 38, at 26-27. 

58. Sutton, supra note 10, at.4. 


68. Sutton, supra note 10, at 4. 
69. Id. 
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designate the day of the week (1, 2, or 3).’° The “day” was divided into 
three parts so the third number designated what part of the class day (1, 
2, or 3).” Here is an example of the three parts of one equivalent class 
period: the second week, second day of the three day class week, and the 
first part of that day would be indexed as 2.2.1; the second part of that day 
would be 2.2.2 and then, the last third of that class day would be 2.2.3.” 
Each of these parts would identify a folder that contained three items: 
reading, video-lecture, and quiz.”? 

The distribution for course credit was divided as follows: Quizzes 
(30%); E-Journal (10%); Class and group exercises (10%); Midterm 
(10%) and Final examination (25%); and the Video project (15%).”* The 
objective for this division of credit was to include a number of different 
assessment methods,’* compare them to a traditional course and to ensure 
that no one assessment method would dominate the points available for 
the course, like a traditional law course where all or most of the credit is 
based on a final examination or a final paper.’° 


IV. DESIGNING A SURVEY TO ANSWER THE COMPARATIVE QUESTIONS 


Online courses are typically in three categories: synchronous, asyn- 
chronous, or hybrid.” These terms are not defined in the ABA Standards 
and hybrid does not appear at all.”° First, the design of the course was 
asynchronous, making the video-lectures available on-demand, through- 
out the course.’? The quizzes that correspond with the video-lectures were 
open only one week at a time.®° This is essential to keep the class at the 
same point in the course in order to participate meaningfully in discuss 
boards and group exercises. It also ensures that students do not fall behind 
in an online course. This process was administered by opening the week 
at midnight each week, on Sunday, and closing the last week’s work as- 
signments at the same time.*! This ensured that no one would fall behind, 


70. fd. 

71. Id. 

72. Id. 

73. Sutton, supra note 11, at 4. 

74. Id. at 5, 

75. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW SCHS. § 314 at 23 (AM. 
Bar ASs’N 2019). 

76. Sutton, supra note 11, at 5. 

77. STANDARDS AND RULES OF PROCEDURE FOR APPROVAL OF LAW SCHS. § 306 at 19 (AM. 
BAR Ass’N 2019). 

78. Id. 

79. Sutton, supra note 11, at 5. 

80. Ia. 

81. Id. 


2020] Asynchronous, E-Learning in Legal Education 153 


and discussion board exercises could be done where everyone had com- 
pleted the same work during the week.* Second, the course was designed 
for students located anywhere, so no face-to-face sessions or even online, 
interactive sessions were conducted during the course.** This absence of 
face-to-face contact with the professor was intentionally designed into the 
study to test the comparative aspects of a course with absolutely no face- 
to-face contact with the professor compared to the traditional, in class 
experience.** So no meetings were held in-person with students for pur- 
poses of this research.*° 


V. PROCESS OF CONDUCTING THE RESEARCH 


The survey was approved for human subject testing by my Univer- 
sity’s Institutional Review Board.** Students were asked to complete the 
survey after the course work had been completed for the course.” They 
were offered a link to another free, online course, if they completed the 
survey.*® Three follow-up reminders were sent to non-respondents, and 
at the end of the survey period of four days, 100% of those surveyed, 
responded.*° Not all surveys were completed, with only thirty to thirty- 
two respondents completing almost all of the questions, resulting in 75% 
substantial completion.” Because the cohort was so small (thirty to 
thirty-two), primary data and qualitative measurements were collected to 
indicate trends rather than conclusive findings of their perceptions.” 

The survey methods used ranking on a scale of one to five, one being 
the lowest; and five being the highest when asked comparative questions 
about traditional courses compared to this asynchronous e-learning 
course.” Questions were asked about format, time spent learning, com- 
prehension, learning styles, assessment methods, feedback to the learner, 
convenience, and learning habits.” In addition, demographic questions 
as well as indirect interest questions about course interests were used to 


82. dd. 
83. Id. 
84. Sutton, supra note 11, at 5. 
85. ld. 
86. Id. 
87. Id. 
88. Id. 
89. Sutton, supra note. 11, at 5. 
90. ld. 
91. Ia. 
92. Id. 
93. Ta. 
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identify any biases in self-selection for the e-learning course.”* 
VI. THE FINDINGS OF THE STUDY 


A. Demographics 


Demographically, the students were in their second and third years 
of law school, all were in the first semester of each of those years, since 
we have a traditional program and it was the fall semester (2015). There 
were also four graduate engineering students in the course.°° There was a 
relatively equal balance in gender with 49.7% female and 51.3% male, 
which reflects law school enrollment.’ The age distribution was typical 
of law school and graduate school.’ There were 38.7% in the eighteen to 
twenty-four age category; 58.06% in the twenty-five to thirty-four age 
category; and 3.23% in the forty-five to fifty-four age category.” Only 
thirty-one of the forty-two respondents completed the age question.'” 
This was the first time any of the students had taken an online, asynchro- 
nous course in law school.'°' I retested the students in the course in fall 
2016, and only a year later, 16.7% had previously taken an online course 
in law school. 


B. Comparisons to a Traditional Law Course 


The students were asked a series of comparative questions: “How 
would you rank this course in comparison to other law courses you have 
taken? Answer with one being the least or lowest and five being more or 
most. If the comparison is the same, then you would rank it, three.” 

For most of the responses, there was a close to normal distribution, 
bell-curve, from one to five. The following comparisons elicited re- 
sponses with a normal distribution: Level of difficulty; time required to 
do the quizzes; time to prepare for class; your percentage of “attendance”; 
and your understanding of the material.'!°* This indicated for these factors, 
most students found no significant difference between the e-learning 
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course and the traditional course.'” 

There were two questions designed with a predicted distribution 
skewed to the low end, but the results for this comparison were surpris- 
ing: “How regular was the interaction with the professor compared to 
other courses?”!”° Considering the students did not once meet with the 
professor, it was predictable that this would result in a distribution solidly 
on the one to two end of the distribution, but it was surprisingly distrib- 
uted with almost half of the respondents ranking this response with a 
three to five.'°’ One of the tradeoffs with an e-learning course is frequent 
e-contact through the E-Journal with a weekly response from the profes- 
sor in addition to graded written assignments, and responses to other as- 
signments.!°* Also, email contact was available “24/7” with a promised 
response within twenty-four hours.’ At least half of the respondents 
found “regular interaction with the professor” to be equivalent, and this 
may have been one of the most surprising findings, given they were never 
face-to-face with the professor.'!® This suggests students consider online 
contact to be equivalent to face-to-face contact at least in defining “inter- 
action.” 

The second question, which follows the first, “[h]ow was the quality 
of the interaction with your professor compared to other courses?” also 
yielded a surprising result, where 56% of the students ranked this ques- 
tion between three to five.''' This was a surprisingly high ranking of the 
quality of the “interaction with your professor compared to other law 
courses” considering that the interaction was completely online.'!* This 
may also reflect a shift in student perception of interaction finding online 
interaction to be equivalent to face-to-face interaction.'!? 

Predictably, when asked about the, “[c]onvenience compared to 
other law courses?” 94% ranked this question between three to five.''* 
This was not surprising, since the flexibility and asynchronous aspects of 
the course are designed to make it as convenient as possible in ways that 
traditional courses cannot be.'!° 
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A question which was designed to reflect the exact wording of ABA 
Standard 306(d)(2)''°, “[h]ow well do you feel regular monitoring of stu- 
dent effort and communication about that effort was achieved in this 
course compared to other law courses?” was a measure of whether this 
standard was met in this course format.''’ The result was a skewed dis- 
tribution to the upper end (36.7% ranking this question four; and 73% 
ranking this question three to five).''® Technologically, it is possible for 
professors to follow the progress of students, get instantaneous feedback 
on quiz performance and monitor when and for how long they view the 
video-lectures.''® Also, if a student did not achieve a minimal score on a 
quiz, they were blocked from going to the next one without contacting 
the professor to discuss the reasons for the low score.'”° The skewed pos- 
itive distribution could be attributed to this technologically increased 
monitoring and contact.'”! 

The question, “[h]ow satisfied were you with this course compared 
to other law courses”, surprisingly resulted in a normal distribution, sug- 
gesting that the respondents were just as satisfied with this course as they 
were with any traditional law course.'*? 


C. Study Behaviors with Asynchronous Law E-Learning 


The next series of questions explore the comparative learning behav- 
iors with asynchronous, e-learning in law with that of studying in tradi- 
tional courses.'*? The questions were as follows: how they studied, where 
they studied, whether they studied alone or with others, and whether they 
studied with noise or with quiet, and what devices did they use to access 
the course as well as self-assessment of their learning styles.'** 

When asked where they studied (Figure 3)'*°, the choices were: in 
complete silence with no one around and no interruptions; in a relatively 
quiet place with some interruptions; in a noisy place, like a coffee shop; 
while babysitting my children or others’ children; in the library; with 
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headphones, earbuds, etc.'*° The students were asked to identify where 
they studied based on a continuum of always, sometimes or rarely.'*” The 
surprising result was that 90% studied in complete silence with no one 
around and no interruptions, always or sometimes; and 80% responded 
that they studied in a relatively quiet place with some interruptions, al- 
ways or sometimes.: The most surprising result from this set of ques- 
tions was that 93% said they rarely study in a noisy place, like a cof- 
feeshop.'”’ It is notable that 13.8% sometimes do coursework while 
babysitting.'*° 

This raises other possibilities: Could the use of the library for study 
be declining, with 56% saying they rarely do their coursework in the li- 
brary?! Yet, 40% responded that they sometimes did their coursework 
in the librarv.'*? 


aw would you describe your usual way of completing the weekly, 
worktor the course? _ i p 
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In designing the course, the recommended process for completing 
the week’s work, for each of the nine, weekly video-lectures is as follows: 
first, read the assignment; second, watch the video-lecture; and third, take 
the quiz.'*? Repeat for all nine video-lectures.'** This order was merely 
recommended, and there was no technological way to force this order, 
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given the limitations of the Blackboard® platform.'** So after receiving 
some indications from the data reflecting their online activity, that stu- 
dents were watching the video-lectures while at the same time, answering 
the quiz questions, a question was formulated, with some surprising re- 
sults.'3° Self-reporting, 70% of students said that they “watched the video 
lecture while taking the quiz.”'*’ While this is was not exactly the design 
plan for the course, it did result in active watching and listening to the 
video-lectures.!3* Here is the resulting table in Figure 4.'°° 


= 


G8: Now i wanito ask you about the quizzes and how you approached 

these assesemenis. Pick ite response thatis closest te how you 
seomnleted.the quizzes: None of these answers are wrong or contrary to 
any required instructions for completing the work. 
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Whether they studied alone or with others was also a study behavior 
explored in this survey.'*° When asked if they had ever worked together 
with another person in this course, 43% said never; 43% said once and 
13% said twice, while no one said more than twice.'*' The question in- 
cluded the caveat that none of these answers were wrong or contrary to 
any course instructions.! 

There is also folk psychology that suggests students do their work on 
their phone or tablet, which this survey sought to prove or disprove.'*’ The 
question asked what device they used.'** The result showed that 93.7% 
used a laptop to complete their coursework in this course. Only 3% used 
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asmartpad, and 3% used “other’’.'*° This may be due to the fact that Black- 
board® did not yet have quizzes available to be taken on a smartphone. 
There were some anecdotal remarks that some students “mirrored” their 
device on a larger television screen to watch the video-lectures.'4° Indi- 
rect data was also collected from the use of YouTube viewing with regard 
to the device used to watch the video-lectures collected from YouTube 
and was consistent with this self-report. '*’ 

Further study behavior data was collected from indirect measure- 
ments from group video-lecture watching.'** The average time for watch- 
ing a video-lecture for courses in all disciplines is six minutes.'*? The av- 
erage time for watching video-lectures in this course was nine minutes.!*° 
Indications from this data show that only one student (not specifically 
identified) turned on closed-captioning most of the time.'*! 

Another notable observation with the video-lecture watching was a 
spike in watching, creating a peak time for watching pattern.!°? Each 
week, the course material changed at 11:59 p.m., each Sunday, at mid- 
night, and all the previous week’s material closing and the next week’s 
material would become available.'°? The peak work time was always on 
Sunday, several hours before the deadline for the week.'*4 This under- 
scores the importance of requiring short periods of time, e.g., a week, for 
completion of the week’s work to avoid situations where students might 
otherwise fall behind and try to catch up with an entire course in the last 
days of the semester, which is simply technically impossible to do. This 
constraint, designed into the course, on a week by week basis, ensures 
that a student is never more than seven days behind the rest of the class, 
at most.'*° The video-lecture viewing minutes timeline screenshot, shown 
in Figure 5,'°° demonstrates the consistent peaks of activity each Sunday 
before the deadline for the first four weeks of the semester, which con- 
tinued in the same pattern throughout the course.'*’ 
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Figure 5 


Finally, a combination of indirect data and self-reported data was 
used to examine whether there was an expected correlation between 
minutes watching the video-lectures with scores on the quizzes.'°* The 
prediction was that the number of minutes watched will correlate posi- 
tively with the quiz score.'** The indirect data showed only whether the 
video had been opened (not the number of minutes watched), but correlat- 
ing that data with the average quiz score data through “eye-balling” the 
graphic (not a statistically significant correlation) below shows there is a 
correlation with at least opening the video and the score on the quiz.“ 
Without better data on number of minutes watched, this is only an indi- 
cation that this hypothesis warrants further testing. 
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Finally, in another self-report question, when asked to describe 
whether they had learning styles that were auditory, visual or kinesthetic, 
the results were that 56.25% said they were visual learners; 21.88% said 
they were auditory learners and 21.88% said they were kinesthetic learn- 
ers.'! This might encourage the use of resources for all of these learning 
styles, which can all be done with an e-learning, asynchronous format. 


D. Evaluating Assessment Methods 


As previously stated, one of the goals of this course was to use many 
assessment methods in order to evaluate as many as possible for e-learn- 
ing.'® The use of several forms of assessments is also required to meet 
the ABA Standard 314 for using formative assessment methods, as well 
as summative assessment methods.!™ 

In this series of questions, students were asked which assessment 
methods they found the most useful in this course.'®* They were asked to 
rank their usefulness on a continuum, with five different values: useless 
(1), had some use (2), acceptably useful (3), very useful (4), extremely 
useful (5).!® Predictably, there was a normal distribution around most of 
the assessment methods, and each discussion board, each project was 
specifically named in the survey.'® Figure 7 shows how similarly they 
rated the assessment methods, but there were some notable differences.!®’ 

Interestingly, the 115 quizzes throughout the course, received a 
weighted average of 3.00, a higher score than I had predicted.'® The E- 
Journal was the lowest weighted average of 2.00, which I had not pre- 
dicted.'® I later learned that feedback for the E-Journal was hidden in a 
way that made it difficult for students to see it, and thus many students did 
not receive feedback. This may have attributed to the lower score for the 
E-Journal. The video-project scored the highest weighted average at 
3.16. 

The video-project was an interactive tool for developing an idea 
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throughout the semester and then making a six-minute video of the stu- 
dent explaining legal issues around the student’s self-selected (but ap- 
proved) topic.'’' The assignment was to select an emerging technology in 
the first weeks of the course and apply areas of law covered in the course 
throughout the semester, culminating in a video analysis.” This is com- 
parable to the traditional oral presentation or legal topic in-class presen- 
tation. The other part of the project is that once the videos are all posted, 
every student watches and comments on the video presentation of each 
student in the class. '”? The watching time for forty-two student-videos 
was equivalent to about one week of class time. Although only by a small 
amount, the video-project was ranked as the most valuable assessment 
tool by the students.!”* 


Overall. as demonstrated in Figure 7. the differences between the 
75 


E. Anecdotal Observations 


One opportunity for purely anecdotal observation was where stu- 
dents sought technical support from the research librarian faculty.” In 
my discussions with the research librarian faculty who assisted these stu- 
dents, they reported that a recurring comment was that their perception 
was that an online course would be easier than a traditional course, but the 
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work was as much or more than a traditional course.'”’ The survey elicited 
anormal distribution for the “Level of Difficulty” question compared to 
other law courses.'’* One possible explanation might be that the anecdotal 
responses indicate the students needing technical support may have found 
it more difficult simply because of the technology. 


F. Biases 


Demographically, the near equal gender distribution should avoid 
any gender bias.!” By self-selection of this course, Emerging Technol- 
ogies Law, it is possible that only the students with an interest in tech- 
nology or skilled in the use of online technologies, might have biased the 
positive responses to the course.'®® To answer that question, the survey 
included a question that was designed to determine if the students were 
interested in primarily technology type courses or was there a mix of stu- 
dents who were not particularly interested in technology law courses.'*! 
The question that was asked was whether the respondent had an interest 
in any of the other technology-related courses, without identifying the in- 
structor, whether it was online or traditional, time it was taught, or any 
other information that would alter their response to anything other than 
the subject matter indicated by the course title.'8 They were asked, “[a]re 
you interested in taking any of these emerging technology courses, as tra- 
ditional or e-learning courses? Check all that apply.”'*? The choices were: 
Law and Biotechnology; Nanotechnology Law and Policy; Cybersecurity 
Law and Policy; Space Law; Intellectual Property Law; Patent Law; and 
Global Biosecurity Law.'** The highest response was 57% on any one 
course, and all courses fell between 33-57% responding that they would 
be interested.'®° 

This suggests that not more than half of the class was generally in- 
terested in technology and law courses; in fact, it probably represents a 
normal distribution, and so technology-savvy or technology-interested 
students were not over-represented, so no bias is indicated for this factor. 
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CONCLUSION 


The need for data in this area of teaching law in an asynchronous, e- 
learning format is great, given the interest in all disciplines and among all 
universities to make more teaching content available online. Despite the 
overwhelming majority of law schools still cautiously watching, but with 
a trend toward increasing online learning, more data could accelerate the 
development of this activity for law schools, if there was empirical data 
regarding the efficacy of the methods in comparison to traditional law 
courses.'*° I wrote in 2016, that I hoped that this study can be the start of 
opening inquiry in this field and exploring more questions as well as those 
that have been raised by this study.'®’ I am pleased to report that another 
study is forthcoming regarding online learning for law students.'** 

The findings in this study are promising for asynchronous law e- 
learning, and should replace folk psychology regarding some of the find- 
ings. Compliance with the ABA Standards tested here, 314 and 306 were 
found to have been met with this asynchronous, e-learning platform. Pos- 
itive aspects of asynchronous, law e-learning, tested here showed that un- 
derstanding the material, preparation time, level of difficulty; and surpris- 
ingly, contact with the professor and meaningful contact with the 
professor were only slightly skewed toward less than the traditional 
course, but close to a normal distribution for “comparable” to traditional 
courses. '®® Convenience, compared to traditional courses was strongly in- 
dicated as “more so” by 94% in the comparison to traditional courses. !”° 

Methods of working on coursework yielded some surprising results 
that crack the myth that students are working on smartpads and 
smartphones, finding that 90% of the students used their laptop for 
coursework.!®! Further, perhaps surprisingly, 90% of students “almost 
always” and “sometimes” study in quiet places with no interruptions, as 
opposed to noisy places like coffee shops where a smaller percentage of 
the students responded that they rarely used these places to do course- 
work.!? The majority of students “rarely” used the library, perhaps indi- 
cating a need to reconfigure libraries for private e-learning spaces.'”° 
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In terms of the value students placed on various assessment methods 
in the course, the students most enjoyed the interactive assessment meth- 
ods, like the video project which they created and then watched and then 
commented on other student videos, online.'** The discussion board top- 
ics were also favored with a slightly higher ranking.” 

With regard to learning styles, they self-identified approximately two 
to one, as visual learners; whereas the remainder were equally split as 
self-identified auditory or kinesthetic learners.'° This is useful in plan- 
ning a combination of assessment methods throughout the course. 


A. Hybrid Courses Compared to Asynchronous Courses 


There is a great interest in hybrid courses, and one of the questions 
I received about this research when it was first made available was from 
a law school interested in hybrid courses and how they compare to asyn- 
chronous courses. Based on this question, I used the small sample of stu- 
dents who had taken both an asynchronous course followed by a hybrid 
course. The three students took the same asynchronous course and the 
same hybrid course. These questions were asked in the Spring 2016 se- 
mester, and because of the small sample size, are purely anecdotal but 
contain some interesting insights that are useful. 

Demographically, three students identified as male and one student 
identified as female. Three out of four of the students preferred the hybrid 
course format. Those that liked the hybrid form more cited more collab- 
oration, accountability, contact with the professor, and discussion time as 
the reasons for their preference. 

They were asked questions comparing the workload and the experi- 
ence. Here are a few of the comments: 


I like cyber security because you see the professor and you are held 
accountable. 


If you do not know the material, people will know. 
Much more student collaboration and application of material. 


There was more interaction with the professor in Cybersecurity Law 
[hybrid], since we were able to meet with her weekly. .. . Online stu- 
dents often expect the professor to be available much more often via 
email and to provide quick feedback/answers, as they lack the face-to- 
face classes and interaction. 


This was my favorite thing about cybersecurity [hybrid course]. We got 
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to listen to the lectures and get a full understanding of the law at a time 
that fit within our schedule. Then we got to go to class and apply that 
law and hear other people’s view on how they would apply the law. For 
me, the hybrid e-learning classes are basically the best of both worlds. 
I got the flexibility of the asynchronous course without sacrificing the 
discussion and interaction that you get with a face-to-face course. 


The hybrid e-learning course, may offer the best of all worlds, and 
the combination of online lectures with face-to-face discussions may in- 
dicate this is the optimum way to use e-learning in law schools. The re- 
cent approval of the hybrid J.D. degree in a growing number of law 
schools could offer that combination that is the best of both worlds. More 
study is needed for the hybrid courses and the hybrid J.D. degree. 


B. Recommendations 


For future asynchronous, law e-learning courses, which are planned, 
this research yields the first empirical insights that might be used in the 
design of these courses for optimum benefit to the student, as well as 
satisfaction with meeting the learning objectives and compliance with 
ABA Standards. 

For assessment tools, incorporating more interactive assessment 
tools like the video-project which scored highest as a valuable assessment 
tool is another recommendation for course design. Finding new and crea- 
tive ways to add more interactive elements to the course design, is another 
possible indication. 

The hybrid e-course is also emerging as a favored form of teaching 
law, and data collected in this study of asynchronous course design and 
teaching can also be equally useful in the design of the hybrid course. 

Finally, the course used in this study was designed to be tested as an 
asynchronous, law e-learning course with no face-to-face contact with 
the students, only online contact, asynchronously. But for communica- 
tion, the use of synchronous meetings online, could provide an opportunity 
for resolution of any questions, while making the rest of the course asyn- 
chronous. While this 100% asynchronous law, e-learning course, proved 
to be comparable to a traditional course, overall; for future courses that 
are predominately asynchronous, it might make for a better transition for 
faculty and students into the world of asynchrony, to use synchronous 
online meetings, perhaps monthly or bi-weekly, to ensure broader oppor- 
tunities to include synchronous communication with students. But as 
more students become accustomed to online learning, the face-to-face 
meeting or synchronous meeting may become unnecessary. 


